The invention relates generally to video image display and to deinterlacing video images.
Many television and video signals are interlaced, where the set of scan lines (typically 525 for NTSC color television) which make up a single video frame are not scanned or transmitted sequentially. Rather, the video frame is divided into two xe2x80x9cfieldsxe2x80x9d, each field comprising every other scan line. In television, the scan lines comprising one field are transmitted first, followed by the scan lines of the second field.
However, a number of display devices, such as computer monitors, are not interlaced. Rather, these devices sequentially scan the entire display area, one scan line after another. To display an interlaced scanned sequence, such as a video signal, on such progressively scanned devices, a deinterlacing process must convert each separate field into a complete display frame that can be sequentially output to the display device. The main task of a deinterlacing process is to reconstruct the missing line between each of the scan lines of an interlaced field.
There are two primary deinterlacing methods, each with their own strengths and weaknesses. xe2x80x9cInter-fieldxe2x80x9d techniques simply merge the data from the second field with the data from the first field to produce a completed frame. If there is no motion in the video frame, such methods yield an ideal reconstituted picture. Vertical resolution can be as good as an original noninterlaced frame. However, if there is motion within the video signal, motion effects will generally be visible to the human eye. Motion effects arise when an object, which was in one location during the scanning of the first field, has moved when the alternating scan lines of the second field are scanned. Simply combining the interlaced scan lines of the two fields yields an unacceptable rendition of the object.
xe2x80x9cIntra-fieldxe2x80x9d techniques use data only from a single field to produce a complete frame. Such methods are better suited for video frames having motion. With an intra-field technique, the values for non-existent pixels are interpolated from pixel values in the scan lines above and below the non-existent pixels. While this technique produces no deleterious motion effect, since it does not incorporate motion from one field to the next, it also does not enhance vertical resolution, since it merely interpolates from existing pixel values within a single field and does not use pixel information for missing scan lines from the second field. Also, simple intra-field deinterlacing techniques (such as simple vertical interpolation) tend to generate unacceptable jagged pictures along diagonal edges.
In general, in one aspect, the invention features a method for interpolating a pixel during the deinterlacing of a video signal, the video signal including at least two fields of interlaced scan lines, each scan line including a series of pixels having respective intensity values, the method including generating a motion value representative of the motion between successive frames about the pixel, detecting an edge direction about the pixel, performing an edge adaptive interpolation at the pixel, using the detected edge direction, and performing a motion adaptive interpolation at the pixel, using the generated motion value.
Embodiments of the invention may include one or more of the following features. The motion value can be generated by comparing segments of pixels about the pixel from successive frames. The segments can include at least two consecutive pixels in a scan line. Generating the motion value can include, for each of a plurality of pixel segments from successive frames about the pixel, calculating a plurality of differences, determining the greatest of the calculated plurality of differences, and determining the motion value from a look-up table using the greatest of the calculated plurality of differences. Whether an edge exists about the pixel can be determined and the look-up table selected from a plurality of look-up tables depending upon whether an edge exists. The motion value can be generated by comparing segments of pixels about the pixel from at least three successive frames. The pixel segments for at least one of the successive frames can be stored in a feature buffer.
The edge direction can be detected by forming the respective differences for a first plurality of pixel pairs, each pixel pair comprising two pixels from scan lines respectively above and below the pixel, each pixel pair for the first plurality of pixel pairs having a first common direction, calculating a first combined difference for the first plurality of pixel pairs from the respective differences for the first plurality of pixel pairs, forming the respective differences for a second plurality of pixel pairs, each pixel pair comprising two pixels from scan lines respectively above and below the pixel, each pixel pair for the second plurality of pixel pairs having a second common direction, calculating a second combined difference for the second plurality of pixel pairs from the respective differences for the second plurality of pixel pairs, and comparing the difference between the second combined difference and the first combined difference with an edge threshold to determine whether an edge exists.
Each of the first and second plurality of pixel pairs can include at least three pixel pairs. The calculating of the first and second combined differences can include adding the absolute values of each of the respective differences formed from each of the respective pixel pairs for the first and second plurality of pixel pairs, respectively. The first direction can be along an approximate 135 degree angle from vertical at the pixel and the second direction can be along an approximate 45 degree angle from vertical at the pixel.
The edge direction detection can further include forming the respective differences for a third plurality of pixel pairs, each pixel pair comprising two pixels from scan lines respectively above and below the pixel, each pixel pair for the third plurality of pixel pairs having a third common direction, calculating a third combined difference for the third plurality of pixel pairs from the respective differences for the third plurality of pixel pairs, and if an edge exists, comparing the third combined difference with the first and second combined differences to determine the edge direction. Each of the third plurality of pixel pairs can include at least three pixel pairs. The calculating of the third combined difference can include adding the absolute values of each of the respective differences formed from each of the respective pixel pairs for the third plurality of pixel pairs. The calculating of the third combined difference can further include multiplying the added absolute values with a sensitivity factor. The third direction can be substantially vertical.
The edge adaptive interpolation at the pixel can further include, if the edge direction is substantially vertical, then forming an intermediate pixel value by averaging at least one pixel substantially directly above and at least one pixel substantially directly below the pixel, if the edge direction is approximately 45 degrees from vertical, then forming the intermediate pixel value by averaging at least one pixel above and at least one pixel below the pixel and substantially along a 45 degree axis through the pixel, and if the edge direction is approximately 135 degrees from vertical, then forming the intermediate pixel value by averaging at least one pixel above and at least one pixel below the pixel and substantially along a 135 degree axis through the pixel.
The motion adaptive interpolation at the pixel can be performed by multiplying the intermediate pixel value by the motion value and adding the value of the pixel from the next field multiplied by the difference between a maximum motion value and the motion value to form a final interpolated value for the pixel.
In general, in another aspect, the invention features apparatus for interpolating a pixel during the deinterlacing of a video signal, the video signal including at least two fields of interlaced scan lines, each scan line including a series of pixels having respective intensity values, the apparatus including a motion value generator configured to generate a motion value representative of the motion between successive frames about the pixel, an edge direction detector configured to detect an edge direction about the pixel, an edge adaptive interpolator configured to perform an edge adaptive interpolation at the pixel, using the detected edge direction, and a motion adaptive interpolator configured to perform a motion adaptive interpolation at the pixel, using the generated motion value.
Embodiments of the invention may include one or more of the following features. The motion value generator can compare segments of pixels from successive frames about the pixel to generate the motion value. The segments can include at least two consecutive pixels in a scan line. The motion value generator can further include a difference calculator configured to calculate a plurality of differences for each of a plurality of pixel segments from successive frames about the pixel, a difference comparator configured to determine the greatest of the calculated plurality of differences, and a look-up table of motion value configured to be indexed by the greatest of the calculated plurality of differences from the difference comparator. The motion value generator can further include an edge detector configured to detect whether an edge exists about the pixel, where the look-up table is selected from a plurality of look-up tables depending upon whether an exists. The motion value can be generated by comparing segments of pixels about the pixel from at least three successive frames. A feature buffer can store the pixel segments for at least one of the successive frames.
The edge direction detector can further include a first pixel pair comparator configured to form the respective differences for a first plurality of pixel pairs, each pixel pair comprising two pixels from scan lines respectively above and below the pixel, each pixel pair for the first plurality of pixel pairs having a first common direction, a first pixel pair difference calculator configured to calculate a first combined difference for the first plurality of pixel pairs from the respective differences for the first plurality of pixel pairs, a second pixel pair comparator configured to form the respective differences for a second plurality of pixel pairs, each pixel pair comprising two pixels from scan lines respectively above and below the pixel, each pixel pair for the second plurality of pixel pairs having a second common direction, a second pixel pair difference calculator configured to calculate a second combined difference for the second plurality of pixel pairs from the respective differences for the second plurality of pixel pairs, and an edge existence logic configured to compare the difference between the second combined difference and the first combined difference with an edge threshold to determine whether an edge exists.
Each of the first and second plurality of pixel pairs can include at least three pixel pairs. The calculating of the first and second combined differences can include adding the absolute values of each of the respective differences formed from each of the respective pixel pairs for the first and second plurality of pixel pairs, respectively. The first direction can be along an approximate 135 degree angle from vertical at the pixel and the second direction can be along an approximate 45 degree angle from vertical at the pixel.
The edge direction detector can further include a third pixel pair comparator configured to form the respective differences for a third plurality of pixel pairs, each pixel pair comprising two pixels from scan lines respectively above and below the pixel, each pixel pair for the third plurality of pixel pairs having a third common direction, a third pixel pair difference calculator configured to calculate a third combined difference for the third plurality of pixel pairs from the respective differences for the third plurality of pixel pairs, and an edge direction calculator configured to compare, if an edge exists, the third combined difference with the first and second combined differences to determine the edge direction. The third plurality of pixel pairs can include at least three pixel pairs. The calculating of the third combined difference can include adding the absolute values of each of the respective differences formed from each of the respective pixel pairs for the third plurality of pixel pairs. The calculating of the third combined difference can further include multiplying the added absolute values with a sensitivity factor. The third direction can be substantially vertical.
The edge adaptive interpolator can further include logic configured to, if the edge direction is substantially vertical, form an intermediate pixel value by averaging at least one pixel substantially directly above and at least one pixel substantially directly below the pixel, if the edge direction is approximately 45 degrees from vertical, form the intermediate pixel value by averaging at least one pixel above and at least one pixel below the pixel and substantially along a 45 degree axis through the pixel, and if the edge direction is approximately 135 degrees from vertical, form the intermediate pixel value by averaging at least one pixel above and at least one pixel below the pixel and substantially along a 135 degree axis through the pixel.
The motion adaptive interpolator can include logic to multiply the intermediate pixel value by the motion value and to add the value of the pixel from the next field multiplied by the difference between a maximum motion value and the motion value to form a final interpolated value for the pixel.
Advantages of the invention may include one or more of the following. By smoothly combining both intra-field and inter-field values for each pixel in a frame, weighted by the detected motion at the pixel, the invention can provide for greater accuracy in deinterlacing scanned video signals and greater fidelity in the displayed image, for both static and dynamic portions of scene. Use of inter-field interpolation techniques for pixels having lower motion values allows for higher vertical resolution. Use of intra-field interpolation for pixels with higher motion values reduces unwanted motion effects. Combining both in a linear function, weighted by the detected motion at the pixel, can smoothly provide the best of both techniques. By selecting a motion value look-up table based upon whether an edge is detected at the pixel allows for more accurate motion value determinations. Determining motion values based upon segments of pixels also can increase motion detection accuracy and reduce noise and false motion determinations. Further, since each pixel in a segment will share a common motion value, segmentation sharply reduces the number of motion calculations required per frame. Use of at least three frames of segmented pixel information allows for more accurate motion determinations. Use of segmentation and feature buffers for storing segments of prior frames can reduce the bandwidth required to perform feature buffer reads and writes. It can also eliminate the need to perform segmentation calculations more than once per frame. Interpolating a pixel value along a detected edge provides further accuracy in deinterlacing a video signal.
These and other features and advantages of the present invention will become more apparent from the following description, drawings, and claims.